An Introduction to Graph Compression Techniques for In-memory Graph Computation

In this work we attempt to answer the following question: How large a graph can we process using a vertex-centric model of computation in the main memory of a single machine? Specifically, we use a modified Pregel framework to calculate PageRank, identify connected components, and single source shortest path algorithms on two large graphs. While it is not possible to load these graphs into memory with naive representations, we show that by using well known graph compression techniques, we can not only load the graphs but also run vertex centric programs on them, even on machines with fairly limited memory. We evaluate both adjacency list and adjacency matrix graph compression. For high-degree vertices in an adjacency list, we show a space savings of around 70%, while for all vertices in the graphs we saved around 45%. Using a compressed adjacency matrix representation we saved around 40% for all vertices – high-degree vertices could not be compressed further because of extra data associated with these vertices. After compressing the LiveJournal graph, which has around five million vertices, we were able to run the PageRank algorithm in approximately 10 seconds per superstep, while the shortest path algorithm ran in 8 minutes per superstep.